Abstract. The 9-Neumann operator on (0, q r )-forms (1 < q < n) on a bounded convex 
domain O in C n is compact if and only if the boundary of fl contains no complex analytic 
(equivalently: affine) variety of dimension greater than or equal to q. 
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, 1. Introduction. 

Let O denote a bounded pseudoconvex domain in C n . For 1 < q < n, let L 2 Q g )(0) denote 

the space of (0, g)-forms with square integrable coefficients, with the norm || ^ aidz~i\\ 2 
= Jo \ a i\ 2 dV(z), where the prime indicates the summation over strictly increasing 
(/-tuples. The 9-Neumann operator N q is the (bounded) inverse of the (unbounded) self- 
adjoint, surjective operator dd* + d*B. We refer the reader to [FK], [Ko], [Kr2], and the 
recent survey [BS3] for background on the <9-Neumann problem. 

Compactness of the 9-Neumann problem is a basic property with many useful conse- 
quences. In the case of domains with smooth boundary, it implies global regularity of the 
d- Neumann problem (in the sense of preservation of the L 2 -Sobolev spaces), see [KN]. Also, 
the Fredholm theory for Toeplitz operators is a direct consequence of the compactness of 
the <9-Neumann problem ([V], [HI]). In a related context, whether or not the 9-Neumann 
problem is compact has ramifications for certain C*-algebras of operators naturally asso- 
ciated with a domain in C n ; compare for example [Sa]. 

Catlin [Ca2] proved compactness of the 9-Neumann operator on smoothly bounded do- 
mains whose boundary satisfies Property (P). The boundary of a domain O satisfies Prop- 
erty (P) if for every positive number M there is a plurisubharmonic function A G C°°(Q) 
with < A < 1, such that for all z G 60 and w G C n , p =1 (d 2 \(z)/dz a dzi3)w a Wi3 > 
M\w\ 2 ([Ca2]). Property (P) was studied systematically under the name of S-regularity in 
[Si] (in the context of arbitrary compact sets in C n ). It was recently observed that Catlin's 
result remains true when no boundary smoothness at all is assumed: the 9-Neumann prob- 
lem is compact on a bounded pseudoconvex domain whose boundary is a S-regular set 
([St]). (Under the additional assumption that the domain is hyper convex, compactness of 
the d- Neumann problem had been shown earlier in [HI].) 
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In this article, we discuss compactness of the <9-Neumann problem on bounded convex 
domains. We obtain a complete characterization of compactness by the absence from the 
boundary of complex analytic ( equivalent ly: afhne) varieties of appropriate dimensions. A 
closely related (in fact, in the context of convex domains, equivalent) question is that of 
compact solution operators for 3. For 1 < q < n, consider 3 as an unbounded operator from 
£( g _i)(^) to L 2 g )Hker 3. A bounded linear operator S q from L 2 ^nker 3 to L 2 q _^(^l) 
is called a solution operator for 3 on (0, g)-forms if dS q u = u for all u G L 2 ^ fl ker 3. 

The following terminology will be convenient: an affine variety of dimension q is a 
(relatively) open subset of a complex affine subspace of C n of dimension q. 

Theorem 1.1. Let Vt be a bounded convex domain in C n . Let 1 < q < n. The following 
are equivalent: 

(1) There exists a compact solution operator for d on (0, q) -forms. 

(2) The boundary of Vt does not contain any affine variety of dimension greater than or 
equal to q. 

(3) The boundary of Q does not contain any analytic variety of dimension greater than 
or equal to q. 

(4) The 3 -Neumann operator N q is compact. 

The implication (4) =^ (1) holds in general (and is well known): compactness of the 
3- Neumann operator implies compactness of the canonical solution operator. In fact, 
the formula N q = (3* N q )* (3* N q ) + (d*N q+1 )(d*N q+1 )* (see [R], [FK]) shows that N q is 
compact if and only if the canonical solution operators d*N q and d*N q+ i are compact. 
Note that in statement (1), it is the same to say that the canonical solution operator is 
compact, since compactness is preserved by projection onto the orthogonal complement of 
the kernel of 3. 

Henkin and Iordan [HI] recently showed that (on a bounded convex domain) the d- 
Neumann operators are compact for 1 < q < n if there are no one-dimensional analytic 
varieties contained in the boundary of the domain (see [Si] for the smooth case). It has 
been known for some time that analytic discs in the boundary of a smooth pseudoconvex 
domain in C 2 obstruct compactness of the 9-Neumann operator. Specific examples for 
failure of compactness are given in [Li] and [Kr2]. These examples are pseudoconvex 
Reinhardt domains in C n . Theorem 5 in [Sa] implies that on a pseudoconvex Reinhardt 
domain, compactness of the canonical solution operator on (0, l)-forms is incompatible with 
analytic discs on the boundary. To what extent analytic varieties in the boundary obstruct 
compactness of the 9-Neumann problem on "general" domains seems to be open. On the 
other hand, it is known that obstructions to compactness of the <9-Neumann problem can 
be more subtle than analytic varieties in the boundary (so that the above characterization 
is false without some assumption on the domain): Matheos [Mt] has recently shown that 
there exist smooth bounded pseudoconvex Hartogs domains in C 2 without discs in their 
boundaries, whose <9-Neumann operators are nonetheless not compact. 

We remark that we make no explicit assumption on smoothness of the boundary. How- 
ever, convexity implies that the boundary is Lipschitz (see e.g. [Mz], §1.1.8). 

The remainder of the paper is organized as follows. In Section 2, we briefly discuss the 
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equivalence of conditions (2) and (3) in Theorem 1.1. This may be part of the folklore, 
but we include the (simple) argument for completeness. Section 3 contains the proof that 
(2) implies (4). The proof of Theorem 1.1 is completed in Section 4, where we show that 
(1) implies (2). We conclude the paper with some additional remarks in Section 5. 

Acknowledgement: The authors thank Harold Boas for stimulating conversations. 

2. Varieties in the Boundary of a Convex Domain. 

In this section we show that (2) implies (3) in Theorem 1.1 ((3)=^ (2) is trivial). This 
may be viewed as the simplest manifestation of the general principle that on the boundaries 
of convex domains, questions of orders of contact with analytic varieties are determined by 
the orders of contact with affine subspaces ([Mc], [BS2], [Y]). Related observations may be 
found in [N], [Ch]. Complex manifolds in general (but smooth) pseudoconvex boundaries 
are studied in [BF]. 

We first observe the following: if V is a (/-dimensional variety in C n , then its convex 
hull V contains an affine variety of dimension q. This is clear if n = 1. For general n, the 
observation follows by induction on the dimensions as follows. If V has non-empty interior 
(in C n ), we are done. If the interior of V is empty, V is contained in a real hyperplane (since 
then there are no 2n line segments with end points in V which are linearly independent 
over M). After a suitable change of coordinates, this hyperplane is {x n = 0}. By the open 
mapping property of non-constant holomorphic functions, applied to the restriction of the 
function z n to (the regular part of) V, V is contained in the complex hyperplane {z n = 0}. 
This completes the induction. 

To prove that (2) implies (3) in Theorem 1.1, assume now that bQ contains a q- 
dimensional analytic variety V. Let po be a regular point of V so that near po, V is 
a (/-dimensional complex manifold, and assume without loss of generality that a support- 
ing hyperplane for O at po is given by {x n = 0}. The argument in the previous para- 
graph shows that if V Po is the intersection of V with a small neighborhood of po, then 
Vp Q ^ {z n = 0}. Consequently, the convex hull of V Po is likewise contained both in O and 
in {z n = 0} C {x n = 0}, hence in 60. But by the above observation, this convex hull 
contains a (/-dimensional affine variety. 

3. Sufficient Conditions for Compactness of N q . 

Proposition 3.1. Let Q be a bounded pseudoconvex domain in C n . Assume that for every 
positive number M , there exists a neighborhood U of bVt and a C 2 -smooth function A on 
U, < A < 1, such that for all z G U, the sum of any q (equivalently: the smallest 
q) eigenvalues of the Hermitian form (d 2 X(z) / 1 dzadzp) 7 ^^^ is at least M. Then the d- 
Neumann operator N q on O is compact. 

Note that for q = 1, the above condition on the Hessian of A reduces to the condi- 
tion that appears in Property (P). For q > 1, this condition does not imply that A is 
p/nrisubharmonic (while it still implies that A is subharmonic) . 
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To prove Proposition 3.1, we first note (following [H], p. 137) that the condition on A 
implies 



for all z G U and f <E A z , where A 2 ,q ' denotes the space of (0, g)-forms at z, and the 
prime indicates summation over increasing (q — l)-tuples I. (3.1) can be seen by using 
a frame (at z) where the Hessian of A is diagonalized. The proof of Proposition 3.1 is 
now based on [Ca3], Theorem 2.1 (see also [BS3], Section 2, for a somewhat different 
approach to this type of estimate) and [Ca2], proof of Theorem 1. The fact that no 
boundary smoothness is assumed necessitates working on smooth subdomains and using a 
regularization procedure for the forms involved that was introduced in [St] . The details of 
this argument are carried out in [St], proof of Corollary 3, to which we refer the reader. 

In order to show that (2)=>(4) in Theorem 1.1, it now suffices to show that if the bound- 
ary of a bounded convex domain contains no affine varieties of dimension q or greater, then 
the assumption in Proposition 3.1 is satisfied. This can be done by suitably generalizing 
the arguments in [Si], Proposition 2.4 that cover the case q = 1: the Choquet theory 
has to be done for a cone of functions that reflects the condition on the Hessian used in 
Proposition 3.1 (rather than for the cone of plurisubharmonic functions). We also need a 
substitute for the fact that boundary points of a convex domain O are peak points for the 
algebra of functions holomorphic on O and continuous on O when there are no varieties of 
positive dimension in the boundary of O. We now develop the necessary ideas. 

For a compact set X in C n , denote by C(X) the usual algebra of continuous functions 
on X, with sup-norm. Denote by H(X) the closure in C(X) of the functions holomorphic 
in a neighborhood of X. A closed subset E of X is said to be a peak set if there exists 
/ G H(X) such that f(z) = 1 on E and \ f(z)\ < 1 on X \ E. The function / is called a 
(weak) peak function on E. 

Proposition 3.2. Let X be a compact convex subset ofC n , let zq G X , and let 1 < q < n. 
Then there exists a complex affine subspace L of dimension < q — 1 through zq such that 
X fl L is a peak set if and only if X contains no affine variety of dimension > q through 

Zq. 

Proof. The =>- direction follows easily from the maximum modulus principle. To prove the 
reverse direction, we need Glicksberg's peak set theorem (cf. [G2], pp. 58) which says that 
E is a peak set if and only if ve G H{X)~ l for all finite regular Borel measures v G H{X)- L , 
where ve is the restriction of v to E. 

We argue by induction. The case n = 1 is clear (but see the case q = n below). 
Assume the conclusion for n — 1, (n > 2). We need to establish it for n. Since there 
is no affine variety of dimension > q through zq and contained in X, z$ is a boundary 
point. Without loss of generality, assume that z$ is the origin and X C {Rez n > 0}. Set 
g(z) = exp(— yfz~^) (where the square root is the principal branch). Let J = X fl {z n = 0}. 
If q = n, let L = {z n = 0}. Then J is a peak set: the function g(z) is a peak function. 



(3.1) 
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Now assume that 1 < q < n — 1. Then J is a compact convex subset of C n_1 , and by 
the induction assumption, there is a complex affine subspace L C {z n = 0} = C n_1 of 
dimension < q — 1 such that JflL is a peak set for H(J). We now show that XC\L = J(lL 
is a peak set for H(X). Let v be a finite regular Borel measure on X, v G H{X)^. Thus we 
have for any holomorphic polynomial / and positive integer m that f x f ■ g m dv = (note 
that g, although not itself analytic in a neighborhood of X, is in H(X)). Letting m — > oo, 
we obtain that Jj f dv = 0. The convex set J is polynomially convex, so the holomorphic 
polynomials are dense in H( J) by the Oka- Weil approximation theorem. Consequently, 
vj G H(J) 1 - and hence vjnL £ Hi^J)- 1 -, by Glicksberg's theorem. Using Glicksberg's 
theorem in the other direction, we conclude that J fl L is a peak set for H(X). This 
completes the induction and the proof of Proposition 3.2. 

For an open set U C C n , denote by P q (U) the set of continuous functions A on U such 
that for any z £ U and orthonormal set of vectors {t±, • ■ ■ , t q } in C n , the function 

C = (Ci, • • • , C g ) e C 9 ^ a(z + Ci*i + • • • + C«*«) 

is subharmonic on {C e C 9 ; z + Ci^i + • • • + ( q t q G C/}. That is, P q (U) consists of the 
continuous functions on U that are subharmonic on each (/-dimensional complex affine 
subspace. In particular, P\(U) is the set of all continuous plurisubharmonic functions and 
P n (U) is the set of all continuous subharmonic functions. P q (U) is a convex cone in C(U) 
that is closed under taking the pointwise maximum of finitely many of its elements. Note 
that each function in P q (U) is a locally uniform limit of C°°-smooth elements in P q of 
slightly smaller open sets: this follows from the usual mollifier argument. Finally, it is not 
hard to check that - £J=} \z \ 2 + (q - 1) YTj= q \h? ^ ^(C n ). 

We now return to a bounded pseudoconvex domain O. We denote by P q (bQ) the closure 
in C(bQ) of functions that are in P q in a neighborhood of 60. A probability measure \i on 
60 is said to be a P^-measure for z G 60 if 

(3.2) \{z) < [ Xd^ XeP q (bQ). 

JbQ. 

We refer the reader to [Gl], Chapter 1 for a treatment of these measures in an abstract con- 
text, and for the elements of Choquet theory. In particular, P q {bVt) satisfies the properties 
(1.1)-(1.3) in [Gl]. _ _ ' ^ ' _ , ^ ' 

Let O be a bounded convex domain, and zq a boundary point through which there is 
no affine variety of dimension q or higher that is contained in 60. We claim that the only 
Pq-measure for z$ is the point mass at zq. Note that there is also no affine variety of 
dimension > q through zq that is contained in O (this is a special case of the argument at 
the end of Section 2). By Proposition 3.2, there is a complex affine subspace L of dimension 
< q — 1 such that 60 fl L is a peak set for H(Q). Let / be the corresponding (weak) peak 
function. Because / G H(Q), |/| G P q (bQ). (3.2) now shows that any P 9 -measure v for zq 
is supported on 60 fl L. In suitable coordinates, we may assume that zq is the origin and 
L C {z q = . . . = z n = 0}. In (3.2), take now A = - J2jZl l*jf + (?-!) EJ =9 M'- We 
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already know that the support of v is contained in L, where A reduces to — X)j=i \ z j\ 2 ■ 
We thus obtain from (3.2) that the support of v consists of the point zq. 

We next invoke Edwards' theorem ([Gl], Theorem 1.2): for every continuous function 
u on 60 and z G 60, inf{J bf2 ud/i ; \i is a P g -measure for z} = sup{A(z); A G P q (bQ), A < 
u on 60}. Because all P 9 -measures have point support, the theorem gives 

(3.3) u(z) = sup{A(z); A G P q (bQ), A < u on 60} 

for every function it G C(60). For M > 0, let um{z) = —M\z\ 2 . It follows from (3.3) and a 
compactness argument similar to the proof of Dini's theorem that um can be approximated 
uniformly on 60 by functions in P 9 (60), hence by functions that are smooth and in P q in 
a neighborhood of 60. In particular, there exists a neighborhood U of 60 and a function 
A G P q (U)nC 2 (U) such that 0<A + M|z| 2 <lon?7 (after shrinking U if necessary). The 
sum of the smallest q eigenvalues of the Hessian of A + M \ z | 2 is at least qM > M (because 
A G P q (U), the sum of the q smallest eigenvalues of the Hessian of A is no n- negative) . 
Therefore satisfies the assumptions in Proposition 3.1, and the proof that (2) implies 
(4) in Theorem 1.1 is complete. 

4. Necessary Conditions for the Compactness of Solution Operators to 8. 

In this section, we prove the implication (1)=^(2) of Theorem 1.1. One of the main tools 
in the proof is the Ohsawa-Takegoshi extension theorem ([OT], [02]). We also use an idea 
that comes from [Cal] (see also [DP]). 

We first prove an auxiliary lemma. Denote by K^{z,w) the Bergman kernel function 
of a domain 0. 

Lemma 4.1. Let be a bounded convex domain in C n . 
(1) For any po G 60 and p± G 0, there exist constants C > and So > such that 



(4.1) Kq(ps,ps) > CKn(p28,P2s) 

for any 5 G (0,S ), where p 5 = p + 8{p\ -po)/lbi ~ Po\ 
(2) For any sequence {pj} G O converging to po G 60, 



(4.2) lim §feM= = 0, 

~ V K n{Pj,Pj) 



locally uniformly on O. 

Proof. (1) Let U be a ball with center p and radius r the minimum of dist(pi,60) and 
Ibi — Poll/2. Let n = (pi — Po)/\\pi — Poll- It is easy to see from the convexity of O that 
T 5 (0 n U) C O for < 5 < - poll/2, where T 5 (z) = z + Sn. Let 5 = r/2. 
Then for < 5 < S , 



Kq(ps,ps) > CKfi nU (p s ,ps) = CK Ts (ci nU )(p 2 5,P2s) > CKn(p 2 s,P25), 
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where the first inequality follows by localization of the kernel ([JP], Theorem 6.3.5), and 
the last inequality holds because T§ (f2 fl U) C O. 

(2) This part of the lemma is implicit in work of Pflug (see [JP, §7.6]) and Ohsawa [01] on 
the completeness of the Bergman metric. We recall the proof for the reader's convenience. 
Without loss of generality, assume that O contains the origin. It suffices to establish 
pointwise convergence: Vitali's theorem (note that Kq (• , Pj)/y/ Kq (pj , pj ) has norm 1) 
then implies that the convergence is locally uniform. For z G O, let f(w) = Kq(z,w). 
Then ||/(«;) - /(*u;)||n -> as t -> 1". Now fix t, < t < 1. Then 

1/(^)1 < I/Cpj) -/(*p 3 -)I , \f(tPj)\ 

y/K^~p~) ~ y/Kti{pj,Pj) y/K n (pj,Pj) 

<II/W-/Wlla + \ K f; tp '\ 

The domain O is convex and so satisfies an outer cone condition. Therefore, Kn(pj,pj) — > 
oo as j — > oo (see e.g. [JP], Theorem 6.1.17). Thus, letting first j — > oo, then t — > 1~, we 
obtain part (2) of Lemma 4.1. 

We are now in a position to prove the implication (l)=>-(2) in Theorem 1.1. 

Proof of (1)=>(2). Arguing indirectly, we assume that there exists a compact solution 
operator S q on (0, (/)-forms and bfl contains an affine variety of dimension q. (Thus q < 
n — 1.) After an affine transformation, we may assume that {(z 1 , 0) G C n ; \z'\ < 2} C 60, 
where z' = (zi, . . . , z q ). Let z" = (z q+1 , . . . , z n ). 

Let Oi = {z" G C n " 9 , (0,z") G O}. It follows from the convexity of O that fii is 
a (non-empty) convex domain in C n_ «(z"). Let fi 2 = G C n ~ q ; 2z" G fii}. Then 
\z! G C 9 ; l^'l < 1} x fi 2 ^ ^ : every point in this set is the midpoint of a line segment 
joining a point in {l^'l < 2} x {0} to a point in {0} x f^. 

Let po be a point in fi 2 and let pj = po/j,j G N. Let 



y/K^Jp~p~) 



Then = 1- We have 

= 2 -2(n-q) K Cli(Pj,Pj) > c 

for j large enough. The first inequality follows because KQ 1 (pj,pj) < (KQ 2 (pj,pj)) 1 ^ 2 
H-Kfii (•>Pj)||n 2 ( obtained by applying the reproducing property of Kq 2 (pj, •) to the func- 
tion Kq 1 (-,pj)). The last equality follows from the transformation formula of the Bergman 
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kernel. The last inequality follows from (4.1). On the other hand, by (4.2), /_,- — > locally 
uniformly on fii. Consequently, no subsequence of {fj} can converge in L 2 {Vt2)- 

By the Ohsawa-Takegoshi extension theorem [OT] (see also [02]), there exist L 2 - 
holomorphic functions Fj(z', z") on O such that Fj(0,z") = f 3 {z") and \\Fj\\ Q < C. We 
now use an idea from [Cal] (compare also [DP]). Let a.j = Fj(z' ', z")dzi A • • • A dz q . Then 
dctj = 0, || ctj || > ^(n) < C. Let gj = S q aj. Denote by gj the form obtained from gj 

by discarding terms containing a dzj with q + 1 < j < n. For z" G fi 2 fixed, we can 
think of the forms ctj and gj as (0, q) and (0, q — l)-forms respectively, in the variables 
z' = (zi, . . . ,z q ), \z'\ < 1. Note that we still have d z /gj = a^, where d z > denotes d in 
the variables z' . Let (•, •) be the standard pointwise inner product on forms in C q . Let 
X £ Cq°(— 00, 00) be a cut-off function such that 0<x<l,x = l when t < 1/2 and x = 
when t > 3/4. Let (3 = x{\ z '\)dz\ A • • ■ A dz q . It follows from the mean value property of 
holomorphic functions that for z" G 2 , 



Therefore, after integrating in z" , 

ll/i-/fc||n 2 <C\\gj -g k \\ L 2 o q _ i)(n) < C\\gj - gkh^ ^n) as j,k^ 00. 

Since {fj} has no subsequence that converges in L 2 (0 2 ) 5 {dj} has no subsequence that 
converges in L 2 Q g _ 1 ^(0), contradicting the compactness of 5"^. This completes the proof 
that (1) implies (2) in Theorem 1.1. 

5. Further Remarks. 

1) The arguments in Section 4 can be localized by using suitable cut-off functions in 
z" as well; it is enough to control the geometry locally. One then needs a lemma to the 
following effect: Let U\ and U2 be neighborhoods of a boundary point po of a bounded 
pseudoconvex domain O, U\ CC U<i- Then Kq(w,w) and ||i^n(-, w)\\n n u 2 are comparable, 
uniformly for w G fi l~l U\. This can be shown by applying the reproducing property 
of Kn n u 2 to Kq(-,w) and using that Kn, n u 2 (w, w) and Kq,(w,w) are comparable. Also, 
compactness of the <9-Neumann problem is a local property: if every boundary point has 
the property that a compactness estimate holds for forms supported near the point, then 
the 8- Neumann problem is compact. This shows that Theorem 1.1 holds on domains that 
are locally convexifiable. 





gki'&^dV {z) (t? is the formal adjoint of d z >) 
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2) It is noteworthy that in the proof that (1) implies (2) in Theorem 1.1, we have only- 
used that there is a compact solution operator to 8 on the (0, (/)-forms with holomorphic 
coefficients. 

3) On smooth bounded convex domains there is a hierarchy of regularity for the d- 
Neumann problem which can be described in terms of the contact with the boundary of 
affine complex varieties. N q is subelliptic if and only if the order of contact with the 
boundary of (/-dimensional affine complex varieties is bounded from above ([Ca3], [Mc], 
[Y]); N q is compact if and only if the boundary contains no (/-dimensional affine varieties 
(Theorem 1.1); finally, N q is globally regular regardless of whether or not 60 contains 
analytic varieties ([BS1]). 

4) We have stated our results for (0, (/)-forms, rather than (p, g)-forms, as the index p 
plays no role in solving 3. 

5) To prove compactness of N q , we have used (the analogue, for (0, (/)-forms, of) Prop- 
erty (P), see Proposition 3.1. Our work shows that for convex domains, this property is 
actually equivalent to compactness of N q . On general pseudoconvex domains, Property 
(P) still seems to be the only systematic way to derive compactness of the <9-Neumann 
problem, but (to quote from [BS3]) "it is not yet understood how much room there is 
between Property (P) and compactness". 

6) In the proof of Lemma 4.1, we have used the fact that on a bounded convex do- 
main Kq(z,z) — > oo as z — > 60. While this is sufficient for the proof of Lemma 4.1, 
it is interesting to note that for convex domains, there is the (optimal) lower estimate 
Kq(z,z) > C/(dist(z, 60)) 2 . This can be shown by the Ohsawa-Takegoshi extension the- 
orem (see [OT], [02]) and the fact that the estimate is true for bounded convex domains 
in the plane. 
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